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Introduction 

This report documents the Connecticut Department of Higher Education’s efforts to 
investigate and document best practices in assessing student achievement as measured by 
workforce success. That effort is part of a five-state project to Define Best Practices for 
Responsible Accountability Models in Higher Education funded by a U.S. Department of 
Education’s Fund for the Improvement of Postsecondary Education (FIPSE) grant to the 
South Carolina Commission on Higher Education. The original project participants 
included South Carolina, Connecticut, California, Arkansas, and Oklahoma and targeted 
specific outcomes in three areas of concern for higher education: 

1) decreasing college costs; 

2) documenting student achievement including access to higher education, 
achievement while enrolled in college, and workforce success of college 
graduates; and 

3) rebuilding public trust in higher education. 

Each participant agreed to focus on one or more of these outcomes and to act as a peer 
reviewer and collaborator with the other states on their areas of focus. 

Connecticut’s analysis of best practices in documenting workforce success of college 
graduates included the review of twenty-four states’ plans for higher education. 
Identification of states with relevant workforce measures began with the responses to an 
email survey through the State Higher Education Executive Officers (SHEEO) in 
November of 2002. Where SHEEO representatives indicated that employer satisfaction 
surveys were in use in their state, an internet search was performed to locate the state’s 
higher education long-range plans or accountability reports for more details on the survey 
process and purposes. In each case, the plans and reports noted that survey data was 
being used to assess higher education’s contributions to economic development goals. In 
some cases these goals were defined at the state level, at others, they were part of the 
institutional mission. The internet search was then expanded: following leads in 
successive documents as to which additional states and institutions had economic or 
business goals in their higher education plans and what measurements, beside surveys, 
were being used to address these goals. Some surveys and data reports were requested 
directly from institutions and organizations, but many states were not evaluated because 
no statewide plans or references to such plans were found on-line. This may have led to 
some gaps in the data. 

Of the twenty-four plans reviewed, fourteen had some kind of mandate for reporting 
higher education outcomes (from the legislature or from a higher education governing or 
coordinating board) although they varied considerably as to whether all sectors of higher 
education were included and whether the results were aggregated by state or only 
evaluated for each institution or sector individually. Twelve of the 24 had statewide 
goals addressing higher education’s contribution to workforce or economic development 
- many without specific reports or measures defined that would document those 
contributions. Only nine states had mandates, workforce goals and indicators clearly 
lined up to show the contribution of higher education within the state to workforce 
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development. Several were piloting or searching for better indicators or instruments, 
some were still in flux with regard to development of goals for higher education or state 
control thereof. Some had more specific plans in the past that have now been rejected 
based on problems of implementation or interpretation of data. And the most serious gap 
in all the data uncovered, is the fact that even where performance indicators were defined, 
benchmarks were generally not - in most of higher education we are measuring change 
without a clear indication our aim. 

But a few trends have emerged, and a few promising structures and instruments are in 
place. 

An initial reading of the plans and reports available made apparent that four models are 
developing as institutions and organizations discuss measurements of success as they 
relate to business and economic needs: 

1) Reporting graduates and licensure/certification rates by program area. 

2) Determining employment rates of graduates. 

3) Developing qualitative evaluation procedures that utilize direct input from the 
business community to assess graduates or programs with regard to how they 
meet the needs of local business. 

4) Surveying employers or graduate employees on their perceptions of graduates’ 
success in the workplace. 

At some institutions several of these approaches are used as complementary measures 
that together give an overall picture of how well business needs in the state are being met; 
at others the approaches are competing measures as limited resources or concerns about 
data preclude the use of multiple measures. Within each general measurement category, 
there is also some variation as to the actual instruments and procedures that are used to 
assess outcomes, as well as variations in which goals are addressed by which indicators. 



Model 1 : Graduates and Licensure/Certification Rates 

Background 

The justification for using graduates or licensure and certification rates to assess higher 
education’s contribution to workforce success rests on the basic assumption that if 
institutions are producing graduates in areas of employment shortage in the state, they are 
meeting the needs of business. 

Licensure and certification are more often used as measures of student learning goals 1 2 3 
than as indicators for economic development goals 4 . The greatest problem with using 
licensure and certification data for any assessment purpose is that it represents only a 
small percentage of graduates and programs. Even among occupational programs, there 
are only a limited number requiring certification, and among liberal arts programs there is 
much discussion, but no widespread agreement yet, on a skills certification procedure. 
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To address the lack of a credential representing a given set of skills for majors without 
certification requirements, the California Postsecondary Education Commission proposed 
the development of a skills certification instrument which would be part of a “workforce 
report card” for students entering the job market 5 . Under the goal of . .play[ing] a major 
role in the economic development of Tennessee”, Tennessee’s Higher Education 
Commission includes Objective 6.4 to develop “credentialing program” that would 
communicate individual skills and competencies to employers 6 . Both of these plans link 
certification of skills to workforce success. This connection is also the premise of ACT’s 
WorkKeys® three-part program of assessment, job profiling and need determination. 

ACT puts forward their WorkKeys® system as a way to “take the guesswork out of 
hiring decisions by providing reliable measures of a job applicant’s skills” 7 . 

WorkKeys® is currently used by Illinois, Kentucky, Michigan, Mississippi, and 
Tennessee in their statewide workforce development programs. Individual 
programs/institutions in other states are also using it both to evaluate the skills of 
individuals in job training programs, and to match program completers to available jobs. 

The states vary in the locus of both the leadership and provision of workforce 
development programs. Workforce development programs are more often administered 
by state agencies, secondary career and technical education providers and adult education 
providers than by the post-secondary institutions which are the focus of this research. 

The information these programs provide can be useful in the discussion of assessing 
skills as they relate to workforce success, but it should be noted that with regard to the 
broader programs at baccalaureate institutions there is a concern as to whether 
completion of a degree or possession of a set of skills equals actual employment, and 
whether either one equals workforce success in a society where multiple job and career 
changes are the norm. 

Since the mid-eighties, there has been an avalanche of research focusing on business 
leaders’ concerns that the graduates they hire lack the skills to succeed in the workplace 

o 

despite holding a credential . In response to this growing concern, the U.S. Department 
of Labor established the Secretary’s Commission on Achieving Necessary Skills 
(SCANS) for “bridging the gap” between work and school 9 . The result was the definition 
of a set of foundation skills and competencies that respondents found to be needed (and 
lacking) for employees to succeed in the workforce. None were industry specific; they 
were found to be basic across industries and job categories. Although the SCANS report 
focused on the competencies and foundation skills needed for entry-level work and what 
elementary and secondary schools needed to teach to achieve those levels, it also 
postulated that a hierarchy could be developed, in which additional levels within the same 
basic competency areas would correlate to more complex jobs, higher achievement in 
career areas, and presumably, be taught by higher education institutions. 

In 1995, SHEEO and the Education Commission of the States published a review of a 
survey conducted for the New Jersey Business-Higher Education Forum of New Jersey 
employers concerning what they want from higher education graduates and how they 
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view the training services provided to the workforce by higher education 10 . The survey 
used as the point of reference the skills set defined by the SCANS report. 



In the SHEEO survey, participants were asked how important each of the skills were for 
their college-level employees, and how well prepared New Jersey’s college and 
university graduates were in these skills. Along with more traditional academic skills 
such as “reading” and “math” were cross-functional skills such as “problem solving” and 
personal skills such as “honesty”. The report found that while employers rated general 
education skills as important to work, the areas they found most lacking in their graduate 
employees were integrity, honesty, diversity and sociability, not academic skills. 

In another national study of employers, respondents indicated that in the hiring process as 
well they focused less on academics than on personal qualities 11 . In recruiting 
employees, employers ranked an applicant’s attitude during the interview as the most 
important factor in their considerations, followed by their application responses and 
employer references. Much further down on employers’ lists was the importance of tests 
or of school transcripts, which could indicate what academic skills were mastered or at 
least studied in school. A study by the Maryland Business Roundtable for Education also 
confirmed that while business respondents indicate they have difficulty in hiring qualified 
graduates, actual transcripts of potential college hires were not often checked by small 
companies 12 . 



Using Graduates and Certification Rates for Assessing Workforce 
Success 

Currently, Connecticut and New Jersey higher education institutions are reporting 
graduates by program in response to requirements to document contributions to economic 
development. Wyoming utilizes licensure and certification along with other economic 
indicators. 

Connecticut utilizes a multi-tiered system of reporting. The state mandated annual 
accountability reporting and a Performance Task Force was assembled with 
representation from the Board of Governors for Higher Education and each of the four 
sectors of public higher education in the state. The Task Force defined accountability 
goals and common core measures to address them at the state and sector level, and 
specified that mission-driven measures be defined and reported at the institutional level. 

A count of degrees awarded by program area is a common core measure for the goal of 
economic development. All of Connecticut’s accountability measures are reported 
annually. In 2003, trends in graduation were compared to areas of employment shortages 
as reported by the State Department of Labor, but no specific goals or strategies to 
address the shortages were included in the report. 

New Jersey’s Long-Range Plan for Higher Education directs institutions to include in 
their planning processes programs to support New Jersey’s “Major Clusters” (groups of 
interrelated businesses and industries having a major impact on the state’s economy) . 
Again, degree completion relative to industry need in these areas is the performance 
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measure monitored, but no specific benchmarks are cited. New Jersey does have an 
employer satisfaction survey component as a related indicator which is discussed later in 
this report. 

While attainment of credentials is more often used to assess student learning than 
workforce success, the relationship between student learning and workforce success is 
also being explored. South Carolina’s assessment plans link the indicator “support of 
economic development” directly to the goal of improving “Graduate Achievement” 14 . 
The Michigan Department of Career Development used the ACT WorkKeys® program 
in a study to measure the “skills gap”: a statewide measure of the career development 
system’s effectiveness which calculates the difference between the average WorkKeys® 
scores (composite of 4 subtests) of workforce training program participants and the 
average scores of all the job profiles in Michigan 15 . The reduction in this skills gap in 
consecutive years is a measure of workforce development success in the state. 

Currently the National Forum on College-Level Learning is evaluating several different 
measures of student learning (including WorkKeys®) for their utility in assessment 
practices. The forum is an outgrowth of Measuring Up 2000 and 2002 16 , in which the 
National Center for Public Policy and Higher Education compiled available statistics 
relating to higher education performance. The reports present the data as statewide 
aggregate measures in an attempt to foster discussions about benchmarks for such 
measures. Although the College-Level Learning project is focused on measuring student 
learning, not workforce success, Margaret Miller, the Director of the Forum, does equate 
workforce goals to acquisition of general intellectual skills. She notes that one of the 
reasons graduates need to possess the skills being assessed by the project is so that they 
can be successful in the workplace. Conversely, she also writes “In choosing which 
college graduates to hire... industry need[s] information about which skills and abilities 
are certified by a given diploma.” 17 

Utility to Statewide Assessment 

Use of graduation and certification data is fairly straightforward. It is already collected 
by accrediting agencies, although not all licensing and certification boards provide 
national data for comparisons. It has meaning at the state level when analyzed by 
program with relation to state identified areas of industry need, although the relationship 
between program of study (CIP code) and field of employment (SIC code) is not well 
defined. 

The problem of using completions or certification rates for statewide assessment lies in 
the question of what constitutes workforce success at the state level: having a sufficient 
number of skilled graduates or actually filling necessary jobs with those graduates? 

While possession of various skills may ultimately be indicative of success in a job, hiring 
practices (and therefore actual employment of graduates) seem to still rely on the 
intangible results of good interviewing processes. Therefore, numbers of graduates 
produced will not necessarily translate into jobs filled. And if basic skills are, by 
definition, required by all jobs, assessing basic skills does not indicate whether the jobs 
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ultimately filled by these graduates are the ones most needed for the state’s economic 
well-being. 



Model 2: Employment Rates 

Background 

The current most widely used practice for assessing outcomes on business/economic 
goals is tracking employment rates. Seven of the states evaluated were using it for at 
least some sectors of higher education: Connecticut, Florida, Kentucky, Louisiana, 
Oklahoma, South Carolina, and Wyoming. It is a mandatory reporting measure for 
workforce training programs at institutions using WIA funds, in which employment goals 
are set that institutions must meet to maintain funding, and other federal programs such 
as Perkins Vocational Education which provide funds for occupational education. As 
with licensure and certification rates, employment measures are already required for 
occupational program accreditation purposes. 

The justification for using employment data to evaluate workforce development is 
straightforward: the graduates are employed in the workforce. Additional data on 
relationships between programs and fields of employment is necessary, and somewhat 
more difficult to evaluate, if conclusions are to be drawn about how well specific needs 
are being met, or how well specific institutions or programs are doing in meeting these 
needs - but it is possible to look at statewide employment openings by industry and 
graduate employment rates by industry and draw conclusions about higher education’s 
overall contributions to the workforce. 

Using Employment Rates 

The simplest calculation of employment rates involves matching student records to the 
Unemployment Insurance (UI) data base maintained by each state. In Connecticut, the 
State University system and the Community College system both use 6 month post- 
graduate employment rates as performance indicators for the goal of economic 
development. Data is provided by the Connecticut Department of Labor and is reported 
by institution but not broken down by program of study or field of employment. 

Florida’s K-20 Accountability Task Force developed a more detailed model which looks 
at employment, employment retention after 5 years and wage increases . The 
Department of Education maintains the Florida Education and Training Placement 
Information Program (FETPIP) for tracking graduates through the UI wage file and 
federal employee data base. The system includes continuing education data, public 
assistance and state prison data to fully track outcomes for graduates. Information from 
employer satisfaction surveys is also part of the plan for overall outcome assessment, but 
the only link between the two sets of data is the use of the FETPIP data base to identify 
employers to survey. 

Using the FETPIP data base, an analysis was made of employment of Florida’s graduates 
compared to the general population by industry to show where higher education had the 
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greatest impact on the workforce 19 . The results pointed to the complexity of the 
relationship between field of study and area of employment. Besides the difficulty of 
assigning equivalencies between the two, there is significant hiring crossover between the 
two. One common example: a graduate in accounting could show up in the business 
office of any industry. The study cautions that policies designed to increase the 
responsiveness of a program to a specific industry need may actually decrease 
employment options for graduates by narrowing the curriculum scope. This would 
indicate that analysis of how well institutions are serving the overall needs of an industry 
or business is a better option for state level assessment than assessing how well a 
particular program is serving those needs. 

Problems with tracking employment go beyond how to categorize program/industry 
relationships. Those employed in surrounding states and those self-employed are missing 
from a simple search of a state’s UI data base. There is also the problem of assessing the 
contribution of an education to career development through salary increase or promotion 
rather than new job acquisition. Wyoming has addressed these issues through expansion 
of the data base and the follow-up time . Memos of understanding with surrounding 
states allow cross-checking with their UI data bases. An additional 6% to 9% of wage 
data was found in surrounding states. A detailed study by the Wyoming Department of 
Employment, Research and Planning evaluated graduates’ wages as well as employment 
over the first 4 quarters after graduation. This resulted in the conclusion that Community 
College graduates did not always reap the benefits of their degrees in the form of 
increased salary until 12 to 18 months after graduation. The Wyoming study was also 
able to identify a small group of graduates (4.8%) who failed to transition to higher 
salaries or stable jobs. This may ultimately lead to efforts to identify the characteristics 
of this group and develop policies to address their needs. One further finding was that the 
percentage of graduates found in Wyoming and the surrounding states decreased over 
time; a factor which has implications for determining how long after graduation it is 
practical to track students. 

Oklahoma went even further in their analysis by including employment data 5 years after 
graduation . Besides looking at the salary differential among Oklahoma residents with 
and without higher education, the Oklahoma study looked at factors affecting in-state 
retention of graduates in various industries - reasoning that in order to affect state 
economic development the key was retaining qualified graduates 22 . The study also tried 
to improve employment follow-up data by searching for employed graduates who did not 
show up in the UI data base (such as self-employed and federal employees) in the state 
tax commission’s tax return data. This data base only indicates household income and 
does not include specific employment information - there is no indication on joint returns 
as to what proportion of the household salary belongs to the graduate and no job field 
code - but it did account for up to 8% more graduates in the 1998 cohort (percentages 
varied by program) adding to Oklahoma’s understanding of whether graduates were still 
in-state, a key factor in their economic development plans. 
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Utility to Statewide Assessment 

While job acquisition may be a better measure of statewide workforce success than job 
readiness (as measured by graduation totals and certification rates), using employment 
rates introduces a huge number of variables if the goal of state assessment is to formulate 
policies which will effect change at the institutional level. Assessment requires 
consideration of such factors as relation of job to field of study (if no relation, did the 
program of study not qualify the graduate for the appropriate field?), timing of job 
acquisition (before or after beginning a program of study), employment persistence, etc., 
to know whether institutions were effective in preparing students for jobs. Using first job 
acquisition as an indicator of success also skirts the issue raised under graduation and 
certification - that business leaders are dissatisfied with the skills of the graduates they 
hire, so that workforce success is dependent on more than just job acquisition. The 
Wyoming study addresses a number of these issues by extending the follow-up period, 
including wages in the analysis, and also separately surveying employers to evaluate 
satisfaction with employees. 

One advantage of using employment rates as shown in the Florida study, is that when 
combined with labor department information on areas of job shortage, high/low wage 
fields and analysis of emerging industries, this information can provide extremely useful 
data at the state and system level for assessing whether post-secondary institutions are 
producing graduates appropriate to the fields most critical for economic development, 
regardless of from what program they graduated. The data can also be useful to 
institutions in counseling students on what fields will be the most promising for future 
employment. 

As shown by the state models currently in use, much of the employment and wage data 
gathering can be automated. Good, computerized data retrieval makes tracking and 
aggregating the data easier, allowing the long-term analysis and addition of factors such 
as wages which address the concerns about analyzing higher education’s contribution to 
workforce success beyond first hire. The data file maintenance and the analysis can be 
done at the state level, and information provided to institutions can include their own 
service area employment projections. 



Model 3: Qualitative Improvement Processes 

Background 

The third trend primarily involves participation of business representatives in on-campus 
groups evaluating students or programs through capstone courses, portfolio assessment or 
program review procedures. Supporters of this model argue that it is the only one which 
can account for institutional mission and program differences, and for the multi- 
dimensional nature of student outcomes 23 . It also offers local businesses the opportunity 
to specify what skills they need in their employees and gives institutions the opportunity 
to demonstrate where those skills are being incorporated into programs. Creation of a 
process improvement model is, in effect, creation of an action plan to address economic 
goals through program assessment and development. 
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Using Quality Improvement Processes 

There are a number of variations on this model, most relying on institutions to create their 
own processes based on mission and program goals. In Nebraska, the Comprehensive 
Statewide Plan for Postsecondary Education specifies a completely decentralized focus 
with institutionally defined outcomes and measures' . State level accountability involves 
review of institutional ‘portfolios’ some of which contain workforce related factors such 
as job placement rates, narratives on employer partnerships and lists of 
business/community related activities. 

In an effort to have better unified processes and plans, Nevada’s Board of Regents for the 
University and Community College System includes specific statewide goals in their 
Master Plan that institutions must address in their plans in addition to their own 
missions' . Goal 1 is “A prosperous economy” and targets for achieving the goal include 
increasing programs focusing on state needs and critical shortage areas and focusing 
workforce development in areas of high growth potential. Institutions are directed to 
develop a continuous improvement model based on clear mission specification and 
evaluation relative to quality, efficiency and accountability as set out in a RAND study 
done for the state in March 200 1 26 . However, no benchmarks are specified in the Plan, 
and the RAND study primarily focuses on measuring student learning as a means of 
informing a value-added assessment system. 



Utility to Statewide Assessment 

This format is not unlike the process of outside accreditation, and much of the data 
requirements would be the same for the institution. But from the state’s viewpoint, it is 
difficult to evaluate the overall effect on statewide workforce development from program 
assessment data, and if there are many institutions in a state, burdensome to regularly 
evaluate all the portfolios. In addition, to have a true process improvement model, it is 
still necessary to define specific measures to benchmark - whether it is an increase in 
student knowledge, rate of mastery of a (measurable) skill, acquisition of jobs, or 
employer satisfaction with employed graduates. Measures against these benchmarks 
could form the basis for a better statewide analysis, but only if the definition and use of 
indicators is standardized across institutional improvement plans. 

The advantages to this process model approach are generally at the institutional level: the 
depth and immediacy of the information available for program improvement; the 
development of a process-improvement culture at institutions; the public relations 
benefits to involving local business in institutional functions; the chance to make business 
contacts that are advantageous to students seeking employment; and the ability to 
respond to multiple institutional missions and program goals without burdening students 
and faculty with multiple tests. If the state or sector governing agency uses employment 
rates or employer satisfaction as the mandated indicator of workforce success, process 
improvement models can be developed to serve as an effective institutional mechanism 
for increasing those measures and demonstrating responsiveness to needs pinpointed by 
the indicator. Developing those processes could count as progress toward goals in an 
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accountability model even before indicator numbers rise, but they do not eliminate the 
need for a quantitative indicator. 

Model 4: Employer/Employee Satisfaction 

Background 

The final approach to assessing how well institutions meet the needs of business and 
industry relies on post-employment surveys of either employers or the employed 
graduates. Surveys are widely used throughout higher education to track graduates’ 
outcomes for a variety of purposes and evaluating current methods of surveying 
employers was the starting point of this research project. 

Surveys, like employment data, vary as to timing and therefore what question they 
answer. In surveying employers, institutions and governing bodies must agree whether 
the goal is to ascertain the preparation level of incoming employees or the long term 
success of employees. All of the employer surveys found involved skills of recently hired 
employees. In surveying graduates, timing is a factor in another way: there is an effect on 
responses attributable to the respondents’ most recent interaction with the institution. For 
example, if a student finds a job through the placement office of the college, it will 
positively ‘color’ their responses on how their education prepared them for work. It may 
take a few months or even years for them to judge the effectiveness of the actual 
education they received to their workforce success, as opposed to judging the 
effectiveness of the placement office. 

Surveys also vary as to the locus of accountability - is the survey designed to assess the 
institution’s benefit to the graduates they trained or to the businesses that hired them? 
Graduates who are employed may be asked about how their education helped them meet 
their goals, while business leaders evaluate graduates’ skills in relation to the business’ 
needs. These different points of view may lead to different results in answering the 
general question of how institutions are contributing to workforce success. 

The SHEEO survey referred to at the beginning of this report inquired which states were 
requiring employer satisfaction measures in their accountability and assessment reporting 
for graduates of higher education. The responses indicated that in 2002 only a few were 
mandating such measures for assessing graduate success, although more were developing 
such requirements or were aware of campus activities related to assessing employer 
satisfaction that were not specifically mandated. SHEEO respondents also indicated that 
employer satisfaction was more often being assessed for workforce training participants 
than for baccalaureate graduates because customer satisfaction data is currently required 
by the U.S. Department of Labor in performance measures of programs funded under the 
Workforce Investment Act (the proposed 2003 reauthorization act would change this to a 
state-optional reporting requirement). The data collection procedures and reports related 
to WIA activities were also evaluated as potentially useful for assessing workforce 
success of graduates. 
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Using Satisfaction Survey Data 

Surveys referred to in this section may be found in Appendix A. 

Under WIA requirements, both customer (trainee) and employer satisfaction can be 
assessed by the American Customer Satisfaction Index (ACSI). This instrument has long 
been used to assess customer satisfaction with private sector corporations, and has been 
adopted by the federal government for use in improving customer service in a number of 
agencies. Because the questions in the survey are broad, generalized views of a 
customer’s interaction with a program (e.g. “What is your overall satisfaction with the 
services provided?”), they must be asked of a specific population in relation to a specific 
service received to give useful data. Furthermore, the survey is structured not to address 
employer satisfaction with a graduate employee, but with the services provided by the 
institution to the employer. Because of these limitations, in its current form, the ACSI 
does not appear to be useful to an analysis of graduates’ success. 

In Maryland, the state profile of higher education includes overall employer satisfaction 
with colleges and universities in the state based on one question administered by the 
Census Bureau in an employer survey from 1997 . A somewhat more extensive Public 
Agenda survey in 2000 looked at public perception of the graduates, but like the Census 
survey, without an indication of which graduates are being evaluated, it is difficult to 
draw conclusions about the state’s higher education institution’s contribution to those 
perceptions. The results of these types of broad-based surveys may be more useful to 
those studying the issue of building public trust in higher education than in measuring 
workforce success. 

The Maryland Community Colleges have a more extensive survey of their graduates. 
Questions about the employee’s college program’s relationship to their current job as 
well rating the employee’s preparation in 12 skill areas and overall do require that the 
respondent have specific knowledge of the college’s graduates 28 . Most of the data is used 
for internal assessment purposes. The positive response rate from the question on the 
graduates’ “overall preparation for employment” is reported in the colleges’ annual 
Performance Accountability Reports to the Maryland Higher Education Commission as 
their required indicator of employer satisfaction with the colleges’ career program 
graduates. 

Among the Florida state legislature’s mandated measures for institutions are the goals of 
having a “skilled workforce” and “economic development”. These are measured by both 
rate of employment of graduates (see above) and the perceptions of businesses and the 
community that graduates posses the skills they need. Since 1998, the FETPIP data base 
has been used to identify employers of graduates to survey. Each year, several 
occupational areas have been selected for follow-up. 

The survey includes questions about recent employees’ competencies in basic or “soft” 
skills and in specific skills that are unique to the occupation. The report summary" 
cautions that the survey is designed not to evaluate specific graduates, but to evaluate 
overall the skills being taught by the programs. Questions are designed to reflect the 
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“intended outcomes” of each curriculum. For example: education graduates are 
evaluated as to “Plans instruction that draws upon a variety of techniques...” while 
nursing graduates are evaluated as to how well they do “care planning” and engineering 
graduates on their “Ability to design & construct experiments. . .” In these examples, all 
three questions involve planning functions, but lack of comparability between the format 
and even the number of questions asked on each survey form makes it difficult to 
establish a reasonable benchmark score for employer satisfaction. Nor has Florida 
attempted to repeat the surveys by program area on a systematic timetable which would 
allow longitudinal follow-up; making it difficult to assess changes in scores even within a 
single program. The 2003 report focuses on differences in related programs’ satisfaction 
scores for types of institutions (2-year vs. 4-year, public vs. private). 

Illinois, like Connecticut, uses a matrix of state, common and mission specific indicators 
to address the Board of Higher Education’s goal of assisting economic growth in the 

on 

state . They are currently developing a business survey at the state level; there is already 
a common survey of alumni. As of December 2003, the accountability reports were still 
including the results of a 1998 statewide employer survey. Questions on the phone 
survey included overall rating of the preparation of public higher education graduates, 
overall knowledge and skills of graduates; importance of higher education in training for 
particular employer’s needs; and perceived responsiveness of higher education 
institutions to business needs. However, the administration of the 1998 survey was based 
on distribution of industries and business sizes in the state, not the distribution of 
graduates among them. The only determination of whether a business could assess the 
qualifications of public institution graduates was whether college graduates were 
employed by the company at all and, although respondents were told to consider state 
higher education graduates in their responses, there is no check on whether they 
determined who those employees were prior to responding. 

A similar problem was encountered by the University of Texas-Pan American (UTPA) in 
2001 . Their study of employer satisfaction selected businesses to survey by 
determining which businesses had recruited on campus. But after the survey was 
completed, it was determined from the responses that more than 19% of the businesses 
surveyed either did not have any employees from UTPA or (particularly for large 
businesses) could not determine which employees were UTPA graduates. Comments the 
university received from the survey completers included requests that the university 
identify the graduates they wanted evaluated, however, the researchers note that this 
would raise confidentiality problems. 

One improvement in the UTPA survey over the Florida survey was having the same 
questions on skills administered to all respondents, regardless of industry represented. 
This allows overall scores and comparisons to be made. Additionally, respondents 
indicated for various skills both how important the skills were to the job and how 
satisfied they were with UTPA graduates’ performance of those skills. In this way, 
importance and satisfaction are evaluated for the same skill introduced with the same 
wording. This is an excellent tool for program assessment and priorities can be set for 
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program development based on those skills that received both high importance and low 
satisfaction ratings among employers. 

Western Washington University marked its 100 th anniversary in 1998 by doing a multi- 
faceted assessment of how well the university was performing its mission. The 
assessment included a survey of employers of Western’s 5-year-past graduates. Like the 
Florida study, they were able to use an existing data base from the Employment Security 
Department to identify employers of graduates, and confidentiality was maintained in 
how the data were handled and presented. An additional group of employers were 
identified from an alumni survey. Like the UTPA study, Western included an 
opportunity to rate the importance of various skills to their employees along with rating 
performance in those skills. Western’s own innovation was having, for each graduate 
employee, both a human resource manager and a direct supervisor complete similar 
surveys. Although no statistical analysis of reliability was performed, the results of the 
two sets of surveys were remarkably consistent in their overall portrayal of skill 
requirements for various employment areas, and in where the university was doing the 
best and worst in training its graduates in those skills. The summary statistics provided 
both a good “benchmark” of the university’s overall performance in training graduates 
for work, and some clear areas for the various departments to focus on in any program 
improvement plans . 

In dealing with the problem of identifying graduates to employers, the University of 
Nevada in Reno also used a blended approach. In their 1994 survey of employers ~ they 
both contacted businesses that had recruited on campus (asking them to consider UNR 
graduates in their responses, but not identifying any) and contacted alumni for permission 
to survey their employers specifically (assuring the alumni that the university was 
looking to rate the institution not the individual). The researchers noted that while the 
first method leaves uncertainty concerning who, exactly, respondents are considering 
when they complete the survey, the second method leads to self- selection of graduates - 
with the assumption that those who feel their employers are dissatisfied with their 
performance are more likely to refuse to participate. They felt that the combination of the 
two data sources would provide a good data distribution, although in the final analysis, 
they did note a positive response bias. The use of alumni-identified employers was 
dropped in subsequent survey years. 

In the Maryland Community Colleges employers survey discussed above, permission is 
also sought from the graduates to contact their employers and those employers surveyed 
are asked to evaluate the identified employee. There is no analysis of how this selection 
procedure might have affected the survey pool. 

As in models of employment measures, the Wyoming Community Colleges once again 
have one of the most detailed plans for employer survey measures. Originally designed 
to survey graduates of WIA programs, the survey method involves contacting known 
employers of recent graduates and specifically identifying the graduates in their employ 
for them to evaluate when answering the questionnaire. This extensive and intensive 
project was designed and implemented in 2001 by the Wyoming Department of 
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Employment, Research and Planning, using the same UI data base as used for tracking 
graduate employment. The Community Colleges, Workforce Development Council and 
State Youth Council all had input into the project. The identifiable employers of all 2001 
occupational graduates of 4 community colleges were mailed surveys. Privacy was 
maintained because the results were aggregated by program at the state level - no 
individual data was released by the Department of Employment, nor was data presented 
from programs with small numbers of respondents that might allow identification of 
individuals. Both the administration and analysis were at the state level. 

An alternative survey model of asking graduates how well prepared they are for their jobs 
instead of asking employers has also been used to assess how well programs are 
preparing students for work. One such survey of community college graduates was 
found, again, in Maryland. The survey asks only whether graduates are employed and 
whether their degree is related to their employment 34 . These questions appear in similar 
form across many institutions’ graduate survey instruments - data on program/job 
relationship is required by IPEDS for occupational program graduates and therefore 
appears in many states’ community and technical college reports. Allowing graduates to 
assess degree applicability to their jobs bypasses the problem of assigning relationships 
between CIP and SIC codes, and the problem of the limitations of 1:1 program to 
industry assumptions about hiring. However, no study was found which asked graduates 
to assess how well prepared they were in specific skills in relation to the needs of their 
job. In the Wyoming Community College 2001-2002 Performance Report, students who 
transferred to the University of Wyoming were asked about specific skills (e.g. writing 
skills, speaking skills) they wished they were better prepared in prior to transferring - this 
format may have the potential to be expanded to allow graduate employees to rate their 
skills training. 



Utility to Statewide Assessment 

Employer surveys that are not designed to capture employers’ impressions of specific 
employees or institutions are too generalized to be useful for either state or institutional 
accountability or planning purposes. Alternatively, when specific individuals are 
identified for survey respondents, there are problems of confidentiality and 
administration. With surveys of graduates, responses are affected by satisfaction with 
current employment status and overall ties to the graduating institution, but the ability to 
tie outcomes to graduates’ original goals and the ability to track program to job field 
relationships make this data a useful addition to overall assessment and planning - 
particularly where graduates from multiple fields of study are required to meet the needs 
of a particular industry. 

Even with these limitations, data from such surveys can be useful for state and system 
level analyses. For state purposes, it is important to have at least some of the same 
survey questions used across institutions and programs and for administration of the 
survey to be standardized. To facilitate this, the survey could be administered at a system 
or state level to take advantage of economies of scale and to relieve institutions of 
additional reporting requirements - as with employment data, an institution’s results can 
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be provided back to it with additional breakdowns and other data as needed. The larger 
scale of the data collection and reporting makes it easier to protect individual information 
as well. 

If good survey instruments can be agreed upon, and reasonably specific data acquired 
(with regard to who is being evaluated), then reliable, quantifiable data could be available 
for many types of analyses by state, region, industry, field of study, etc. One current 
problem is that none of the surveys have been validated against some other measure of 
employee competence. Several verification models might be useful - a comparison of 
employee and employer ratings on the importance of various skills for a particular job, or 
a comparison of employer ratings of skill importance in a particular job with an 
instrument such as the job skills profile generated by WorkKeys®. A similar comparison 
could be made between employer ratings of satisfaction with employee’s skills to some 
standardized measure of those skills. These types of studies would have to be large 
enough in scale to allow generalizations about the accuracy of the survey instrument. 

Employer or employee satisfaction data as it stands now could be used as one 
performance measure in a process improvement model, adding to the usefulness of 
process improvement to both statewide and institutional analyses. Alternately, if 
employment rate is the performance measure being assessed, employer satisfaction data 
can be used as a check to validate that employment does equate to filling the needs of 
business. Alumni surveys would also act as a check on employment data, especially in 
evaluating whether program of study impacted employment even if the two are seemingly 
unrelated areas. 



Research and Recommendations from Other 
Organizations 

Many organizations are looking at the questions surrounding the accountability of higher 
education to the workforce. Some, like the National Governor’s Association (NGA), 

1C 

approach the question from the point of view enhancing economic development^ , others, 
such as the American Association of Colleges and Universities (AACU) are concerned 
with redefining institutional accreditation 36 - but all discuss measuring outcomes of 
higher education and all include the assumption that part of higher education’s mission is 
preparing workers. As one essay notes, although colleges and universities do more than 
simply prepare students for jobs, “Ours is a society based on work” . Therefore, higher 
education must include in its accountability measures some meaningful estimate of 
higher education’s contribution to this working society. 

The NGA Center for Best Practices urges governors to implement policies to connect 
workforce development to economic needs through education, to build a stronger 
education pipeline partly by funding postsecondary education institutions based on 
demonstrated performance, and to focus on meaningful outcomes including income, skill 
level and job advancement of graduates . This approach argues for a long-term 
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employment follow-up, and uses Florida’s FETPIP tracking system as an example of a 
serious investment in data systems to inform higher education accountability. 

The Southern Regional Educational Board (SREB) also recommends a variety of 
indicators to measure higher education’s performance relative to goals, including using 
graduates’ employment and income information to assess how higher education is 
helping the state respond to changing social and economic conditions . They cite 
certification and licensure rates and employers’ assessments of graduates as program 
measures rather than state measures - although good program assessment is necessary to 
inform good public policy decisions. As was found in the research for this report, SREB 
also notes that after a decade of reporting on higher education, most states have not 
established benchmarks for the indicators they are using. Significantly, Measuring Up, 
the Federal Government’s State by State Report Card for Higher Education, compares 
states on various measures, but sets no benchmarks either - other than establishing a 
national average for various measures. While this data is essential to building a picture of 
higher education - most states had no idea what the distribution of these measures was 
prior to the publication - the report is not a tool for setting improvement goals. 
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Preliminary Findings and Recommendations 

As presented in this background analysis, the discussions, plans and projects for 
evaluating higher education with regard to its workforce and economic contributions are 
legion. As more states are making explicit the link between economic development and 
higher education, the need to assess the contributions of the latter to the former is 
becoming more apparent, as is the lack of information available to do so. Most of the 
states’ long-range plans lack specific measures and benchmarks, or incentives to develop 
them. In many cases, simply adopting business related goals constitutes adequate action 
on addressing economic needs by higher education institutions. 

The elements of a good assessment process are out there: Florida’s and Wyoming’s 
employment tracking systems; Connecticut’s and Illinois’s hierarchies of state, system 
and institutional measures tied in to state-defined goals; New Jersey’s efforts to define 
industry clusters to focus on rather than specific job openings; Nevada’s attempt to put in 
place continuous improvement models informed by value-added measures; several 
institutions’ use of annual employer satisfaction surveys and the efforts to tie employer 
satisfaction surveys more directly to actual employment. Several employer survey forms 
are being used on a regular basis - but they have not been validated against other 
measures of satisfaction or skills. 

Evaluating the data from the studies included in this report, possible next steps in 
developing a comprehensive model of workforce success include: 

1. Discussing the establishment at the state level of an explicit link between higher 
education and economic/workforce development goals. 

2. Using employment outcome as opposed to employment potential (degrees earned, 
licensure and certification granted or skills mastered) as a more direct measure of 
higher education’s contribution to economic development. 

3. Development of an automated employment/education tracking system at the state 
level that allows individuals’ progress from student to employee to be followed. 
Inclusion of longitudinal employment and salary data and access to information 
from surrounding states increases the utility of such a system. 

4. Analysis of graduate employment data in relation to current and projected 
industry needs as supplied by the state’s labor department. 

5. Creation and validation of a survey of employer satisfaction that is linked to 
specific graduates so that the information gathered is less indicative of general 
impressions of higher education and more indicative of how specific programs or 
institutions are addressing industry needs. 
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